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Dear Colleagues: 


Achieving the Dream is pleased to partner with Public Agenda to offer you this important publication. 
Building Institutional Capacity for Data-lnformed Decision Making. The final in a three-part series, this 
guide will help more colleges build institutional research (IR) and information technology (IT) capacity 
and create a culture of evidence in which data and inquiry drive broad-based institutional efforts to 
close achievement gaps and improve student outcomes overall. 

The origin of this series is the recent interim report by MDRC and the Community College Research 
Center, Turning the Tide: Five Years of Achieving the Dream in Community Colleges. The evaluation 
identified areas of great progress as well as aspects of Achieving the Dream's work that need deeper 
focus. Achieving the Dream designed this series with our Founding Partner, Public Agenda, to address 
those recommendations and ensure that every Achieving the Dream Institution has the tools necessary 
to move the needle on student success and completion. This guide advises colleges on how to 
strengthen their IR and IT capacity and provides examples of promising practices at several Achieving 
the Dream Institutions. 

On behalf of the entire team at Achieving the Dream, I'd like to extend my appreciation to Public 
Agenda for their diligent and thoughtful work on this timely series, and my best wishes to each institution 
in pursuit of greater student success outcomes. 

Sincerely, 

H William E. Trueheart 
President & CEO 
Achieving the Dream 



Building Institutional Capacity for Data-lnformed 
Decision Making 


How to Use This Guide 

Building Institutional Capacity for Data-lnformed Decision Making is the third installment of the 
Cutting Edge series, which aims to help colleges engage faculty, scale successful interventions, and 
create a strong culture of evidence through use of data to strengthen their institutional change 
and student success efforts. 

Since data generation and use are at its core. Achieving the Dream has developed a range of 
maferials fo help colleges understand the basic roles of institutional research (IR) and information 
technology (IT) and how to align them with the student success mission at community colleges. 
These resources, available on the Achieving the Dream website 1 , present a wealth of knowledge 
from Data Coaches as well as other evaluation, IR, and IT experts— many of whom were also 
participants in the Public Agenda work group that informed this guide. Rather than duplicate the 
recommendations and guidance, the current work aims to act in concert with earlier publications, 
drawing attention to different aspects of the IR and IT capacity challenge: building institutional 
commitment to data-informed change and facilitating more and better usage of dafa thaf is 
produced. 

Though we begin by presenting a review of the most common challenges and pitfalls to aligning 
capacity in these areas with student success (Section 1), the focus of this guide is on promising 
practices that institutional leaders, IR and IT personnel, and faculty and staff can use at their 
colleges (Sections 2 and 3). Throughout the guide we offer examples of how these practices have 
been applied at community colleges, including detailed examples of two colleges that have 
strengthened their IR capacity through their involvement in Achieving the Dream: El Paso 
Community College and Montgomery County Community College (Section 4). 

In Section 5, we offer colleges using this guide a self-assessment tool for evaluating their institutional 
capacity for dafa-informed decision making. This tool is heavily based on the readiness assessment 
colleges submit to Achieving the Dream when they begin participation; thus, we have named it the 
Augmented Assessment Tool for Achieving the Dream Principle 2: Use of Evidence to Improve 
Policies, Programs, and Services. Rather than use this tool as a grading device, we recommend that 
colleges and leaders use it as a way to prompt careful thinking and thoughtful discussions wifh a 
range of stakeholders from across the college. 

The Appendix includes a list of resources that have been consulted during the course of our work. 
These resources contain a wealth of information and expertise that we encourage colleges to 
access regularly to support their institutional change and student success efforts. 


Achieving the Dream guides available at http://www.achievingthedream.org/CAMPUSSTRATEGIES/GUIDES/default.tp. 


Introduction 


This guide is based on findings from Public 
Agenda's explorafion of the most promising 
practices for increasing institutional research (IR) 
and information technology (IT) capacity at 
community colleges. 

The current study was prompted by an interim 
evaluation report by MDRC and the Community 
College Research Center (CCRC) titled Turning 
the Tide: An Examination of Round i Achieving 
the Dream Colleges' Progress After Five Years in 
the Initiative. As part of an analysis of the early 
experience of the first 26 community colleges 
that have participated in Achieving the Dream 
since 2004 (Round 1 colleges), the report 
observes that the colleges that made the 
greatest progress toward improved student 
achievement shared several key features: broad- 
based engagement of college stakeholders, 
especially adjunct and full-time faculty; 
successful scaling of student success and 
institutional change interventions; and strong IR 
capacity that facilitates the establishment of a 
culture of evidence. The purpose of this third 
guide in the Cutting Edge series is to help more 
colleges build IR and IT capacity that promotes 
the integration of data analyses and inquiry into 
decision making about programs and policies 
that enhance student success. 

The use of data and evidence to improve 
programs and services is a key component to 
any institutional change effort and a core 
principle of the Achieving the Dream model for 
improving student success. Achieving the Dream 
urges colleges to translate this principle into 
practice by aligning their IT and IR capacity with 
the student success mission, establishing 
processes for identifying achievement gaps, as 
well as formulating and evaluating solutions. 
Throughout their participation in Achieving the 
Dream, colleges receive support in the form of 
data coaching from seasoned professionals with 


a working knowledge of using data and 
institutional research to a) identify areas of 
weakness and b) opportunities for 
improvement, and to guide changes in policy 
and practice that lead to better outcomes for 
students. As described in the MDRC interim 
evaluation, however, despite this support and 
guidance, some colleges have struggled to 
build a strong culture of evidence, particularly 
those that began their participation with weak 
data capacity and those facing especially 
strained resources. 

Even amid these and other challenges, a 
number of colleges have demonstrated 
practices that strongly support evidence-based 
decision making. The MDRC report notes a 
number of these specific practices and 
recommends that Achieving the Dream 
investigate promising practices and principles 
to help more colleges succeed. To this end. 
Public Agenda, an Achieving the Dream 
Founding Partner, conducted a multi-method 
study, consisting of a deep literature review, an 
online discussion, and an in-person work group 
of 14 diverse stakeholders, practitioners, and 
experts in IR and higher education reform. The 
work group, facilitated by three Public Agenda 
staff, took place over one evening and one 
day in New York City in June 201 1 and was 
recorded by multiple note takers. This guide 
offers recommendations and insights drawn 
from these sources. 

Meeting the challenge of adequate IR and IT 
capacity goes beyond training capable staff; it 
extends to building institution-wide commitment 
to using data to inform change efforts. 
Consequently, the central question addressed 
by this report is how to build the commitment 
to data-informed change and increase the 
practice of using data effectively to improve 
decision making. 


Given that tight resources present a major 
limitation on what colleges can do to bolster 
their IR and IT capacity, this guide focuses on 
practices that most colleges can apply 
regardless of size, resource availability, 
personnel, or infrastructure. That being said, 
resource constraints have very real impacts 
that cannot be ignored, even as the 
recommendations encourage colleges to think 
beyond their limits. Further, practicing all the 
recommendations contained in the following 
pages will be difficult, even for established, 
well-run, highly productive IR offices. Rather 


than dismiss these challenges, this guide urges 
colleges to consider making strategic decisions 
based on institutional priorities and 
commitments and to think about how they will 
analyze existing resources for potential budget 
reallocation or even external funding to support 
capacity building. As underlined in the MDRC 
report, robust IR and IT capacity is not optional 
if colleges are to reach their student success 
goals. 


What are IR and IT? 

The descriptions of institutional research and information technology functions at community colleges are neither 
one-size-fits-all nor stagnant. Throughout this guide, we encourage readers and practitioners to think beyond the 
traditional functions and the walls of individual departments to find ways to meet their institutions’ data needs to 
inform decision making on behalf of student success. Nevertheless, it may be useful to begin with the traditional 
and basic elements that comprise IR and IT functions at most community colleges. The following are partial 
descriptions of IR and IT in the words of practitioners. 

► Institutional research (which can include research, learning outcomes assessment, planning, institutional 
effectiveness, and accreditation) generally maintains ultimate responsibility for completing all internal and 
external reporting requirements . 2 

► In the context of Achieving the Dream, IR is the locus of activity aimed at assessing student success to help 
colleges decide where to concentrate improvement effort and to measure the impacts of these interventions. 

► IR professionals increasingly perform major work in regional and specialized accreditation activities. IR offices 
are no longer just a place where data is kept but a place where data is converted to actionable information for 
clientele to use for a wide range of purposes, from grant proposals to space utilization. 

► IR is a management process, not a single action, which includes collecting data, analyzing it, and reporting 
information that can be used for decision making. At both ends, this is a people process. 

► IR is the process by which critical decision makers across the college are provided timely guidance for action 
planning based on findings from the analyses of institutional data. 

► Information technology’s role generally focuses on data processing and storage; information systems design, 
operation and maintenance; technical support; and user access and training. Increasingly IT departments are 
also responsible for providing support for instructional technology and campus telecommunication systems . 3 

► IT is heavily engaged in establishing a platform for online instruction and internal communications. 


2 Glover, 2009 

3 Ibid 


Key Obstacles and Common Pitfalls 


Key Obstacles to Building the Commitment to use Student Success 
Data 


► Reporting requirements weigh heavily on colleges. For small and medium-size IR and IT offices in 
particular, the demands of compliance reporting and enrollment tracking duties force staff to de-prioritize 
the kind of analytical work that Achieving the Dream encourages, such as evaluation research and 
responding to faculty or administrative requests for data to support student success efforts. 

► IR and IT functions are not widely visible or understood on many campuses. Not only are IR and IT often 
overloaded by compliance duties; faculty and staff tend to view them in that light rather than as partners in 
institutional change for student success. This tendency is reinforced by a common belief among IR and IT 
personnel that institutional leaders are their only audience, preventing their greater visibility to the college 
community as a whole and faculty in particular. This lack of visibility, in turn, makes it less likely that others 
will trust and make use of their work. 

► Silos between IR and IT and between academic departments complicate coordination. The who, how, 
and when of data access generates tension in many institutions. This is further complicated when there are 
multiple databases in use within a college; for example, academic departments collecting their own data, or 
IR and IT collecting redundant data in separate systems. These systems may not be integrated, and though 
centralized systems are desirable for efficiency and coordination, they are also expensive. 

► Making data accessible to a broad audience is difficult. Data systems can be vast, and data is unclear or not 
tailored for specific audiences; thus, finding information even in a centralized system can require technical 
skills possessed by only a few individuals. 

► Concerns about data integrity inhibit widespread use. Many college data systems are cluttered with 
unreliable data and data entry errors, especially when those entering data lack adequate training or 
supervision. Without resources such as data dictionaries, which describe the contents of a database, and 
validation tables, which define variables, data entered is even more questionable. Faculty, staff, and other 
college stakeholders will resist using data if they believe it is untrustworthy or inaccurate. 
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Common Pitfalls to Building Commitment to use Student Success Data 


► Hiring data people who don’t share student success goals or have good communication skills. In addition 
to possessing essential technical skills, the best data personnel are mission driven, have strong interpersonal 
and broader social scientific skills, and know how to distill and present data in ways that help internal and 
external audiences understand complex issues. 

► Treating data as if it speaks for itself. Staff and administrators can hold a misperception that data reveals 
its implications simply and directly: If the data is positive then something definitely worked; if it’s negative 
then something failed. Data, however, requires interpretation in context and rarely leads directly to simple 
answers. One should not assume that answers will flow effortlessly from data without deliberation and 
careful judgment. 

► Making it impossible for the data to speak at all. When databases are complex, analytic capacity is low, or 
presentations are weak, there is a high chance that data will be misunderstood and misinterpreted. Care on 
all these levels is important to make sure data improves, rather than confuses, your institution’s efforts to 
increase student success. 

► Hiding or ignoring “bad-news” data. When data shows poor program performance or learning outcomes, 
college leaders may be tempted to sweep the bad news under the rug so the institution will not be 
portrayed negatively or to maintain morale among their hardworking, underfunded staff. Bad-news data 
must be handled skillfully, but to put it aside inhibits the very culture of inquiry that Achieving the Dream 
colleges must create to make a difference for their students. 

► Using data as a hammer. On the opposite end of the spectrum is using data to point fingers and assign 
blame. This almost always generates fear and resentment, and it can stifle the formation of a culture of 
inquiry and evidence. 
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Promising Practices for Building Widespread 
Commitment to Data-lnformed Change 


Building a strong commitment to data does not happen overnight, and colleges may be thwarted by 
their prevailing culture— for example, a history of leaders who made decisions based on gut feelings 
rather than evidence; a mind-set in which qualitative data is considered inferior to quantitative data; 
or a buildup of mistrust between faculty and administration that may have originated elsewhere but 
that now thwarts new efforts at data-informed change. Working through such dynamics can be critical 
for building a culture that values quality data and attempts to use it well. 


1. Institutional leaders must set the tone 
of commitment to data 

As noted in the Achieving the Dream framework for 
student success 4 , committed leadership is central to 
establishing a culture of continuous improvement 
that is grounded in inquiry and evidence. 

► Model the Behavior you Espouse. Presidents, 
department heads, and other institutional leaders 
should model behaviors that support a culture of 
evidence and inquiry throughout an institution. 
They should regularly review and explore student 
outcome data with diverse stakeholders in ways 
that spur thoughtful problem solving for student 
success. 

► Connect Data and Vision. Inspire inquiry by 
connecting a commitment to evidence-based 
decision making to the mission of access and 
success. Leadership that is driven by its moral 
obligation to students should be committed to 
facing up to the data in order to identify whether 
the institution is doing all that it can to help 
students achieve their dreams. 


► Use the Hiring Process to Build a Culture of 
Evidence. When hiring for leadership positions, 
make an appreciation for data inquiry and 
continuous improvement a key criterion of 
evaluation. Expectations for data-informed 
decision making should be made clear during the 
interview process 

► Recognize and Support the Champions. Elevate the 
efforts of your campus’s greatest data champions 
and bring experienced and successful exemplars 
and coaches from outside the campus to inform 
and inspire faculty and staff. 

► Walk the Walk Through Resource Allocation. The 
president and board of trustees should commit to 
a culture of evidence and inquiry by allocating 
resources to IR and IT. 


4 Achieving the Dream's framework for student success is based on the premise that to make substantive improvements colleges need to fundamentally change the 
way they operate. Achieving the Dream colleges are encouraged to adhere to four principles: 1) Committed Leadership; 2) Use of Evidence to Improve Programs & 
Services; 3) Broad Engagement; and 4) Action Aimed at Systematic Institutional Improvement. 
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2. Integrate IR and IT into systemic 
student success initiatives 

At many colleges, the IR and IT offices are seen as 
peripheral actors in the effort to retain and graduate 
students. They are not in the classroom, are not 
providing direct student services, and are not making 
decisions that impact a student’s success. This 


philosophical and even physical separation of IR and 
IT from the student-centered mission perpetuates 
silos and limits a broader commitment to elevating 
the importance of data. 

There are a number of practices that can help 
accelerate the integration of IR and IT into the 
student success mission. 


Leadership in Action 

Dr. Alex Johnson, President, Community College of 
Allegheny County, Pittsburgh, Pa. 

When the Coach and Data Coach at the Community 
College of Allegheny County (CCAC) encouraged the 
board of trustees to seek a data-sawy, student-focused 
candidate in the third year of the college’s Achieving 
the Dream participation, Dr. Alex Johnson became the 
new president. For several months after Dr. Johnson’s 
arrival, the data showed poor results in the college’s 
developmental education (dev ed) improvement 
efforts, and not enough faculty were supportive of 
those endeavors. In an effort to turn these results 
around, Dr. Johnson became an active leader, personally 
visiting all campuses and engaging dev ed faculty in 
productive dialogues about student-outcome data, 
seeking their input on possible solutions to improve 
student success. The ideas and energy generated 
through the dialogues were critical to the improvement 
of CCAC’s dev ed success rates, which helped CCAC 
earn the title of Leader College in 2011. Leadership’s 
commitment to using data to improve student success 
was also evidenced by the addition of a qualitative 
researcher to the IR department and Dr. Johnson’s 
regular presentation of qualitative research results to 
college stakeholders. Furthermore, his leadership 
prompted dialogues organized by the college 
assessment director (also the Achieving the Dream 
coordinator) to engage faculty in conversations about 
Community College Survey of Student Engagement 
(CCSSE) and Community College Survey of Faculty 
Engagement (CCSFE) data. As a result of these 
dialogues, faculty developed and helped implement 
plans to remedy the shortcomings in their CCSSE 
outcomes. Overall, Dr. Johnson’s strong leadership and 
collaboration helped the college to institute significant 
changes based on data. 


Dr. Sanford Shugart, President, Valencia 
Community College, Orlando, Fla. 

Valencia College (VC), a Round 1 college, is entering 
its eighth year of participation in Achieving the Dream. 
When the college began its work, President Sandy 
Shugart tied the Achieving the Dream goals to the 
mission of the original charter establishing community 
colleges: to help students succeed. He spoke college- 
wide about the alignment of Achieving the Dream 
efforts with the college’s existing strategic planning and 
accreditation processes. In a call to fulfill the college’s 
mission, he insisted that all stakeholders shift their 
perspectives of the student-college relationship and 
place themselves in the mind of the student. He wanted 
a paradigm shift that would include student-outcome 
data presentations based on the way students experience 
the college, not disjointed reports about student success 
and completion. He also asked stakeholders to embrace 
the philosophy that all students can learn and it is up 
to the stakeholders to help them do so. In addition to 
holding frequent meetings grounded in student- 
outcome data, Dr. Shugart authorized considerable 
college funds to purchase data warehouse servers and 
software to support expanded data access and reporting. 
In so doing, he became a prime example of a leader 
building the commitment to data-informed change. 
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► Use Qualitative Data to Change the Dynamics. IR 
and IT personnel must understand their roles as 
part of the broader mission of increasing student 
success, as well as within the context of 
accreditation. Supplementing quantitative data 
with qualitative data — a key recommendation 
from Achieving the Dream Coaches and Data 
Coaches — helps IR and IT personnel not only to 
expand their own understanding of the student 
experience, but to help others see the connection 
between these roles and student success. 

► Break Down Silos. Seize opportunities to 
integrate IR and IT into reform or planning 
efforts that involve diverse stakeholders. For 
instance, institutional leaders might include IR 
and IT staff in strategic planning. The 
development of online courses offers a prime 
opportunity for instructors, researchers, and IR 
and IT personnel to collaborate on student 
success efforts. Infrastructure renovations and 
hardware or software upgrades are a chance for 
IR and IT to open up new channels of 
communication beyond accreditation and work 
with other staff to enhance student success. 

► Connect the President to the Data. Better 
communication and access between IR, IT, and 
college presidents not only serves the president’s 
decision making; it sets the tone for the rest of 
the organization. This can be facilitated when IR 
and IT leaders report directly to the president 
and her or his top-level administrators. Helping 
build the capacity of college leadership to engage 
data as a vehicle for institutional change should 
be viewed as a vital function of IR and IT, 
pursued with purpose and skill. 

► Make Data Systems More Visible and Accessible. 
Create a clear and workable system for data 
requests, and make sure faculty and staff know it 
exists and how to use it. 


3. Cultivate research leadership geared 
toward student success 

Whether housed within an IR, IT, administrative, or 
academic department, there should be some 
individual or committee that takes ownership of the 
college’s research efforts and, with the support of 
leadership, is empowered to carry them forward. 
Regardless of its composition, research leadership at 
the college should undertake a number of common 
goals: 

► Sharing the student success mission embodied by 
Achieving the Dream and believing in the value 
of data to work toward that mission. 


Mixing Methods: The Importance of 
Pairing Qualitative and Quantitative 
Data 

Qualitative and quantitative research methodologies 
each have their strengths, and most of the time, 
neither type of data alone will answer a question 
fully. For instance, analyses of quantitative data 
often lead to the development of “why” and “how” 
questions that qualitative data can answer best. In 
its turn, qualitative inquiry often generates 
hypotheses that quantitative methods can test with 
greater precision. Therefore, a mixed-methods 
approach is often best. It can: 

► generate and test a grounded theory 

► answer a broader and more complete range of 
research questions, 

► provide stronger evidence for a conclusion 
through convergence and corroboration of 
findings, and 

► produce more complete knowledge necessary 
to inform theory and practice. 5 


5 Johnson and Onwuegbuzie, 2004. 
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► Coordinating data collection and sharing 
arrangements across the college and even among 
different institutions; for instance, between a 
two-year and a four-year school to support 
student transfer. 

► Leveraging student success and institutional data 
to meet various reporting requirements, including 
regional accrediting agencies’ compliance reports. 
The Achieving the Dream report Using Achieving 
the Dream to Meet Accreditation Requirements 
provides practical advice for doing so . 6 

► Facilitating collaboration among IR, IT, and 
faculty researchers to foster effectively 
coordinated systems. 

► Serving as a liaison between IR and IT 
personnel and executive leadership at the college. 

4. Cultivate faculty researchers for 
student success 

Given that many IR departments are already taxed 
by reporting or compliance requirements and are 
understaffed and under-resourced, colleges need to 
find creative ways to fill gaps in analytic capacity and 
ability. One of the richest resources at the 
community college is, of course, the faculty. The first 
publication in the Cutting Edge series, Engaging 
Adjunct and Fidl-Time Faculty in Student Success 
Innovation , includes recommendations for 
institutional leaders and researchers on engaging 
faculty around data and establishing a culture of 
evidence. Here we add to and highlight a number of 
those recommendations. 


► Build on Existing Knowledge. Leverage the 
research and work experience of adjunct and 
full-time faculty to supplement research skills; or 
offer professional development to others in order 
to add to the college’s analytic capacity. 

► Facilitate Departmental and IR Collaborations. 
Establish channels of communication so faculty 
can work with IR personnel and college leaders 
to suggest performance measurement indicators 
based on their firsthand knowledge of their 
students. 


IR/IT Integration in Action 

South Texas College, McAllen, Texas 

South Texas College (STC) began transitioning 
to an integrated data system prior to joining 
Achieving the Dream, but even after converting 
to Banner, an integrated enterprise resource 
planning (ERP) system, the issue of specific data 
owners for the various modules and need-to- 
know data permissions continued to create 
barriers to the data for IR personnel. With 
pressure for more and deeper analyses of data, it 
became evident that IR did need to know all 
institutional data. The chief information officer 
(CIO) agreed that if IR personnel were expected 
to respond to questions regarding institutional 
data, they must be given access to that data. 
Historically at STC there had been no problem 
with IT having access to data, since that 
department is charged with developing and 
maintaining the systems that store it, but the 
college decided that IR needed to be viewed in 
the same way. “You’re one of us!” the CIO told 
the IR director; this comment changed the 
whole picture, as IR and IT began to work 
together to impact student success. 


Manning, 2009. 
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Directly Involve Faculty in Student Success 
Data Analysis. Develop vehicles for IR and 
faculty to collaborate on research that 
aligns with institutional change and student 
success efforts. For instance, faculty can 
help to identify meaningful data elements 
for analysis, provide feedback on 
presentation formats, or colead 
presentations of data to colleagues. 

Support Faculty Innovations and Inquiry. 
Create innovation at the college by 
promoting faculty inquiry and research. 
Many colleges have instituted versions of 
FIGs — faculty inquiry groups, faculty 
interest groups, and faculty innovation 
grants (competitive funds to develop and 
test innovative practices). These structures, 
formal or informal, provide venues for 
faculty to look deeply at institutional data 
and ask critical questions that can help 
move the needle on student success. 

Support Collaborative Action Research (CAR). 
The CAR approach to inquiry involves 
multiple researchers combining their 
expertise and interests toward the common 
goal of changing the practices, policies, and 
knowledge of their shared environment — in 
this case, the community college or the 
department. CAR combines data-informed 
methods with a problem-solving 
orientation to effect real change. 

Provide Appropriate Compensation. Honor 
the time and expertise that faculty 
members contribute to engaging with 
institutional research by providing 
appropriate release time, stipends, and 
public recognition. 


Integration and Research Leadership in 
Action 

Trident Technical College, Charleston, S.C. 

Trident Technical College (TTC) has found a number of 
ways to integrate IR and IT into student success initiatives 
and to share data by developing routine processes for data 
to influence student success, strategic planning, and 
institutional improvement processes. First, TTC aligned its 
Quality Enhancement Plan (QEP) — a Southern 
Association of Colleges and Schools (SACS) accrediting 
requirement — with its Achieving the Dream efforts to 
improve developmental math outcomes. In a show of 
strong leadership, one of TTC ’s Achieving the Dream Core 
Team members, a math faculty member, began holding 
math summits with math faculty, and information from 
these summits was eventually incorporated into their QEP. 
The college received a perfect review from SACS, which 
had zero recommendations for improvement — a rarity for 
that group of accreditors. 

Second, TTC collected considerable qualitative data from 
focus groups and administered nationally validated surveys 
in order to inform its student success efforts. It sent focus 
group facilitators to another Achieving the Dream 
Institution (Orangeburg-Calhoun Technical College), and 
that institution sent its facilitators to conduct focus groups 
at TTC . This experience was a unique sharing and learning 
opportunity, which promoted unbiased facilitation. 

Third, through strong IR leadership, TTC established 
systems to meet the research needs of college stakeholders. 
Not only is the director of IR a talented data presenter, she 
made a smart decision to hire a statistician from the 
college’s faculty for an IR position. This new IR person was 
able to develop a number of analytical and statistically 
validated studies and reports that were useful to 
stakeholders. The number of requests for data increased so 
much that the IR director began working with IT to 
develop an online request system to prioritize and 
document these requests. This is a perfect example of both 
strong research leadership and IR and IT integration. 

Finally, data is now routinely integrated in college 
leadership and annual strategic planning meetings to ensure 
student success. The directors of TTC ’s Achieving the 
Dream effort present data at division meetings and have 
used it to convince faculty to enact significant policy 
changes. As a result, certain students are now required to 
take student success courses, further demonstrating the 
importance of data in improving institutional processes and 
student outcomes. 
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Promising Practices for Facilitating Data Usage 


At many community colleges, the problem is not a shortage but an overabundance of data. The issue 
becomes how to think strategically and analytically about which data matters, how it will be used, 
and who needs to use it. Planning a new program, policy, or initiative that addresses both data needs 
and data analysis and involves key stakeholders can help to give a bigger return on investment. The 
following practices can help with the overload problem and foster a community college culture of 
inquiry into the data. 


1 . Plan for data use, not just data 
collection 

Determining even which data to collect should be 
done mindfully and strategically; then, once 
collected, the raw data needs to be turned into 
information through thoughtful analysis and inquiry. 
For instance: 

► Research leadership must think critically about 
which data elements will produce the most 
meaningful insights to vital questions of student 
success. 

► Track the experience of student cohorts over 
time to help identify problem points along 
students’ path through their education and to 
monitor how institutional changes have 
influenced different cohorts. Achieving the 
Dream Institutions are expected to build their 
capacity for longitudinal cohort analysis; 


Achieving the Dream has provided principles 
and practices of this method in Basics of 
Longitudinal Cohort Analysis. 1 

► Disaggregate cohort and other data by various 
student characteristics (e.g., age, income, gender) 
to get a deeper understanding of students’ 
experiences as well as to identify barriers and 
compare gaps in achievement. Looking at overall 
or average scores can mask disparities in 
achievement. 

► While breaking data down into segments is 
important, it is also essential that data 
presentations keep the full landscape in view by 
putting the numbers in context, thus avoiding 
focusing on small problems that have small 
impact. Further, without a vision of the larger 
context, big-picture issues such as access and 
equity can be obscured. 


7 Voorhes and Lee, 2009. 
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► Emphasize the importance of program evaluation 
data by encouraging IR and the program leads to 
work together to articulate the actions, success 
factors, and intended outcomes of an 
intervention on the front end and then create a 
schedule for testing milestones and impacts in 
concrete ways. Colleges may wish to institute a 
policy of IR sign-off at the beginning of an 
initiative and a series of periodic checkpoints to 
confirm that an appropriate evaluation plan has 
been incorporated. Achieving the Dream 
provides principles and practices of this method 
in Evaluating Student Success Interventions , 8 9 

► Ensure that the various phases of data analysis 
will be conducted with the appropriate tools and 
by those with the most appropriate skills. IR 
personnel with professional skills in analytic 
software such as SAS, SPSS, and Stata can apply 
these tools to explore raw data, while 
administrators and staff might engage only data 
relevant to their immediate questions through 
Microsoft Excel and other spreadsheet 
applications. Leaders should consider investing in 
professional development for those who are 
inclined to learn more advanced analytic 
techniques. The Association of Institutional 
Researchers (AIR) has provided exemplary 
professional development opportunities to IR 
officers from more than 150 colleges through its 
Data and Decisions® Academy. 

► Promote robust qualitative data analysis. As 
noted earlier, qualitative data collected through 
focus groups, surveys with open-ended questions, 
interviews, and observation can help to answer 
questions of “why” and “how” that result from 
quantitative data, and they offer rich fodder for 
important conversations that shed light on 
problems and solutions. 


2. Improve methods of data sharing 
and translation 

The way numbers, analyses, and information are 
translated and presented to various audiences can 
have a profound impact on levels of comfort with 
institutional data. 

► Simplify technical presentations and make them 
audience specific to avoid paralyzing and 
frustrating institutional leaders and other college 
stakeholders. Specific guidance on simplifying 
presentations is offered in the Achieving the 
Dream guide Strengthening Institutional Research 
and Information Technology Capacity through 
Achieving the Dreamd 

► Develop presentations with a representative of 
the group to which the presentation will be 
given. For example, engage faculty leaders in 
creating data presentations that are relevant and 
meaningful to other faculty; pique their curiosity 
and inspire their engagement with the data. 

► Hire IR personnel who can turn data into 
information and then communicate that 
information to others. While technical and 
analytic skills are critical, in addition, strong 
social and communication skills characterize the 
most effective and desirable IR personnel. Where 
these skills do not exist, pursue professional 
development and other opportunities to cultivate 
them. One way to develop these skills is for IR 
staff to copresent with faculty at local, state, and 
national conferences. These kinds of engagements 
give IR personnel experience in managing the 
give-and-take of questions and answers about the 
data, preparing them for similar interactions at 
the college. 


8 Rincones-Gomez, 2009. 

9 Glover, 2009. 
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► Initiate systems that give more people who are 
doing research access to raw data without 
compromising its integrity. For instance, when 
sharing raw data between an IT and an IR 
office, give clear definitions of the data elements 
to reduce the possibility of misinterpretation; 
establish file-freezing policies and make them 
well known; and/or implement role-based 
security or drill-down access so that different 
constituencies in the college have different levels 
of access and can manipulate data accordingly. 


Translation in Action 

Community College of Beaver County, 

Monaca, Pa. 

Dr. Joe Forrester, president of the Community 
College of Beaver County (CCBC), changed the 
institution from one that occasionally reviewed its 
data to one consistently focused on mining data to 
improve performance and presenting that data in 
a concise, easily understood manner. Dr. Forrester 
regularly presents data at college-wide events, 
often with the help of the IR director, Brian 
Hayden, who was hired in part because of his 
excellent presentation skills. In May 2011, the two 
presented the story of their Achieving the Dream 
progress at the National Institute for Staff and 
Organizational Development (NISOD) to strong 
reviews. Hayden regularly presents CCSSE data 
to faculty and engages them in dialogue to ensure 
their comprehension and comfort with the data. 

As a result, the college raised its CCSSE mean 
scores in a single year and was awarded Leader 
College status in August 2011, ultimately 
illustrating that translation and sharing of data are 
crucial factors in ensuring student success. 


3. Create an environment conducive to 
conversations about the meaning and 
implications of data for improving 
student outcomes 

Meaningful dialogue should always accompany the 
presentation of institutional and student success data, 
though the format of the dialogue will vary. More 
and thoughtful discussions among diverse 
stakeholders about what data means can prevent 
common pitfalls, including misinterpretation of data, 
use of data as a hammer, and ignoring “bad-news” or 
uncomfortable data. When faculty or staff discuss 
and analyze student data, their comfort level with it 
increases, leading to more frequent and sophisticated 
use and propelling the college’s student success 
efforts. The following recommendations can help to 
create the right setting and tone for conversations 
about numbers and data analysis. 

► Encourage the president and other executives, in 
addition to IR leaders, to be presenters of data in 
dialogue sessions and college-wide forums. 
Leadership’s presence, knowledge, and ability to 
communicate the data will send a message that 
the data is important and that the college is 
committed to both using data to inform 
decisions and engaging those gathered in 
deciphering its meaning. 

► Reduce fear of data by being transparent about 
how it was collected and analyzed and what will 
be done with it. Providing background in a clear, 
concise, and nontechnical manner up front can 
prevent confusion and skepticism down the line. 
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► Start dialogues with a good question and be 
ready to probe with additional questions. One 
properly worded question can spark lively 
discussion. Framing the conversation with 
questions rather than answers also honors the 
knowledge and experience of the audience. 

► Both college leaders and IR personnel should be 
prepared for and open to questions from 
participants. The use of skilled facilitators, when 
possible, can help the questioning and 
conversation stay on a productive course 


► Build data reflection into existing gatherings and 
consider new opportunities for collaborative data 
inquiry. For instance, utilize faculty orientations, 
professional development days, and department 
meetings as opportunities for data dialogue. 
Convening a data summit can be a strong 
strategy for bringing together a range of 
stakeholders at the college to focus on student 
data. 


Conversations in Action 

South Texas Community College 

South Texas Community College has created an environment that promotes continuous conversations about data 
and its implications for the college’s student success efforts. Making discussion of data a priority has helped the 
college confront the risks of misinterpreting data and failing to use it to inform actions. To avoid these common 
pitfalls, STC developed a research protocol that includes the typical steps for a good study (e.g., literature review, 
appropriate methodology) as well as a step for developing actions. Once the research is available, developing 
recommended actions involves discussion of the findings from the data with the staff or faculty most closely 
associated with the study. For example, if the data is about classroom activities, the faculty or students will be 
convened to discuss the data. This interactivity greatly enhances the researchers’ knowledge of the context of the 
data as well as the quality and feasibility of the recommendations that are offered in a final research report. 

The commitment to having these conversations extends to the leadership level as well. At STC, the president 
regularly reviews data with the IR director and then discusses the data at her administrative meetings. This leads 
to deeper data discussions, follow-up questions, and analyses with lots of back-and-forth between the researchers 
and the administrators. Again, the interactivity can greatly enhance the depth of the study as well as the 
interpretations of the findings. Once specific analyses have gone through this discussion, they are ready to be 
shared with broader audiences by the president, who is well prepared to answer questions about the data. 

Finally, STC has successfully built data discussions into existing gatherings on its campus. For the past seven 
years, the school has convened a community Summit on College Readiness, during which data is provided from 
local, state, and national levels. The summit is heavily attended by educational entities in the region, as well as 
business, industry, and local news agencies. While the first year’s summit was marked by considerable anxiety 
about openly sharing data, over the years, the desire for collaboration — which requires transparency — has 
overtaken the fear of sharing. The region has realized dramatic improvements in the levels of college readiness of 
high school graduates, in part, it is widely believed, due to this summit and the conversations it inspires. 
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Case Studies 


El Paso Community College, El Paso, 
Texas: Faculty Data and Research Team 

El Paso Community College (EPCC) began its 
Achieving the Dream journey with the collection of 
data on how First Time in College (FTIC) students 
performed on their placement exams upon entering 
the college. This data was shared with a variety of 
audiences, including the community Advisory 
Committee, which comprised the superintendent of 
the 12 local school districts, the president of the 
University of Texas at El Paso (UTEP), local 
business and industry representatives, and local press. 
The information was also shared with faculty 
teaching developmental courses in math, English, 
and reading at EPCC, as well as with the rest of the 
college, through town hall meetings held at all five of 
the campuses and the Administrative Service Center. 

As a result of this extensive sharing of FTIC 
placement-exam data, EPCC was able to determine 
that regardless of whether students were entering the 
college directly from high school, less than one year 
after graduating, or later in life, 98 percent were 
placing into developmental courses, especially in 
mathematics. The use of this data allowed the college 
to have very candid conversations with diverse 
constituencies, resulting in the creation of the 
College Readiness Consortium with the local school 
districts and UTEP to reduce the number of 
incoming EPCC students requiring remediation, as 
well as bringing the faculty together for a common 
purpose — to enhance development education in order 
to reduce the length of time students were required 
to be in remediation. 


This positive experience with sharing and using data 
resulted in faculty requesting additional data to 
answer questions about curriculum development that 
they were proposing. This, in turn, led EPCC’s 
president to create an integrated faculty data and 
research team, made up of faculty from the three 
development education areas of mathematics, 

English, and reading, along with representatives from 
the college’s IT, IR, and development areas. The team 
reviews its Achieving the Dream end-of-year self- 
evaluations and continues to identify data needs for 
evaluating its Developmental Education Initiative 
(DEI) as well as other programs. 

Perhaps the greatest accomplishment for EPCC is 
that now faculty members are largerly involved in 
using data and gathering evidence to answer 
increasingly robust questions regarding the success of 
pilots, initiatives, grant-funded activities, and 
interventions implemented in many areas of the 
college. By using and sharing data, integrating IT 
and IR with student success initiatives, and 
encouraging collaboration among various 
departments, EPCC has demonstrated its 
commitment to using institutional research and 
technology to ensure successful student outcomes. 
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Montgomery County Community 
College, Bluebell, Pa.: Engaging the 
College Through Student Success Data 

Montgomery County Community College (MCCC), 
an Achieving the Dream Institution since 2006, 
began its path toward data-informed initiatives in 
2002 with the goal of placing learning first, followed 
by the development of an institutional effectiveness 
model (IEM) to determine where improvements 
were needed. The IEM data includes five-year trends 
and national and state peer benchmarks in student 
retention, persistence, completion, and success in 
developmental and college level courses. When 
MCCC joined Achieving the Dream, data-informed 
decision making had been valued and integrated at 
the leadership level. Since then, through the student 
success initiatives, the culture of evidence has spread 
deeper into the organization. 

To start, MCCC expanded IR and IT resources to 
meet the data and evaluation needs of its staff The 
Achieving the Dream grant initially funded the 
creation of two part-time research analyst positions, 
which have since been institutionalized as a single 
full-time position. In addition, to make data easily 
accessible throughout the organization, IT purchased 
a data warehouse and an end-user reporting tool that 
allows staff to extract their own data, rather than 
relying on IR personnel to access the data for them. 
Key decision makers enjoyed similar benefits, thanks 
to the purchase of data tools with Title III funds. 

The new efficiencies allowed IR personnel to focus 
more time on complex research and analyses. 

Illustrating the importance of data use and sharing 
to foster evidence-based decision making, the 
college’s data team, co-led by the IR director and a 
faculty member, shared disaggregated data on student 
success with the Achieving the Dream Core Team. 

In addition, qualitative data was collected through 
focus groups, helping to identify three key factors in 
student outcomes: 1) developmental mathematics was 
a significant barrier to student success; 2) a 
significant achievement gap existed for African 
American males; and 3) certain subjects should 


become gateway courses. Thanks to these focus 
groups and a clearer understanding of the data they 
had collected, MCCC introduced a series of 
evidence-based interventions, including Concepts of 
Numbers, a redesign of the first developmental math 
course from a topic-based arithmetic approach to a 
conceptual approach; and the Minority Male 
Mentoring Program. 

Continuing with the concept of sharing and 
translating data, the Institutional Research Office 
publishes a monthly newsletter, Research and Practice. 
Each issue is e-mailed to the college community, 
posted on the faculty/staff web portal, and made 
available to the public through the Think Success news 
blog, at http://mc3success.wordpress.com/. 

Finally, MCCC uses data to engage faculty and 
evaluate results of new initiatives. Through the 
Gateway Faculty Academy, faculty members start 
with an initial data packet consisting of charts and 
graphs showing trends in course completion. Analysis 
of this data — in this case, disaggregated by gender, 
race/ethnicity, age, Pell status, and academic readiness 
(determined by placement into one, two or more, or 
no developmental courses) - often leads to additional 
research questions to inform their work. In a shining 
example of collaboration and integration, IR staff 
work hand-in-hand with faculty to help answer those 
questions through additional data analysis and, when 
appropriate, collect data through focus groups. As 
faculty implement interventions, assessment plans are 
developed to ensure learning outcomes are met and 
to evaluate effectiveness. 

As an Achieving the Dream Institution, 

Montgomery County Community College continues 
to dig deeper into its student success data to engage 
the faculty, pilot and evaluate interventions, and 
inform policy changes. Increasing student success 
through data-informed decision making remains an 
anchor for the strategic plan. 
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Augmented Assessment Tool for Achieving the Dream 
Principle 2: Use of Evidence to Improve Policies, 
Programs, and Services 


The Achieving the Dream Self-Readiness 
Assessment has been developed to help stakeholders 
assess the extent to which their college has 
implemented practices that reflect the principles of 
the Achieving the Dream model for increasing 
student success through institutional improvement. 10 
The assessment is designed to help stakeholders 
decide whether Achieving the Dream is a good fit, 
given the college’s existing and desired goals and 
priorities. As stated in the Achieving the Dream 
Field Guide for Student Success: “Colleges are not 
expected to have implemented all of the practices 
listed under the principles. In fact, the Achieving the 
Dream improvement process is designed precisely to 
help Achieving the Dream colleges adopt practices 
like those enumerated in the readiness assessment.” 
The tool is intended to spur thoughtful discussions 
among college administrators, faculty and staff" 
leaders about existing policies and practices, strengths 
and weakness, challenges and opportunities for 
change as they relate to the four principles. Once 
completed, the assessment serves as a launch point 
for a college’s work with the Achieving the Dream 
Coaches and Data Coaches. 


Use of Evidence to Improve Policies, Programs, and 
Services is the second of the Achieving the Dream 
core principles, and the self-assessment asks colleges 
to evaluate their capacity, process for identifying 
achievement gaps, and process for formulating and 
evaluating solutions. In the Augmented Assessment 
Tool below we have revisited the self-assessment tool 
with an eye toward aligning it with the promising 
practices shared in this guide as well enhancing its 
use as a tool for continuous self-assessment 
throughout a college’s involvement in Achieving the 
Dream and beyond. 

We hope that you will use this tool to prompt 
critical thinking and open conversations about the IR 
and IT activities and capacities at your college. We 
have added elements to the existing sections 2.1 - 
2.3 and have added a new section, 2.4, as a way for 
you to consider specifically how you use data to 
inform change. We encourage you to build on this 
self-assessment and use it as a flexible tool that can 
be adapted to your institution’s needs and context. 


10 The four principles of the Achieving the Dream model for increasing student success through institutional improvement are: 1) Committed Leadership, 2) Using 
Evidence to Improve Policies, Programs, and Services, 3) Broad Engagement, and 4) Systemic Institutional Improvement. 
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Principle 2: Use of Evidence to Improve Policies, Programs, and Services 

Extent of Implementation 


Little/None 

Increasing 

A lot 

2.1 

Information Technology (IT) and Institutional Research (IR) capacity 

a 

IT capacity is adequate to meet the demand for data and institutional research aimed at student success. 

i 

2 

3 

4 

5 

b 

Information system or ERP system is integrated or institution has the capacity to easily integrate data from the various 
systems if not integrated. 

i 

2 

3 

4 

S 

c 

Policies and procedures are in place to ensure integrity of data collected. 

i 

2 

3 

4 

5 

a 

IT provides researchers with adequate definitional and operational information regarding data elements and databases. 

i 

2 

3 

4 

5 

e 

IT and IR have developed a protocol for managing data and research requests from college personnel such as administrators 
and faculty. 

i 

2 

3 

4 

5 

f 

IR staff capacity is adequate to meet demand for data collection, analysis, and research aimed at student success. 

i 

2 

3 

4 

5 

g 

IR staff has reporting software and tools to extract and analyze data and produces summary level reports. 

i 

2 

3 

4 

S 

h 

IR staff effectively educates and assists college personnel to understand and use data to improve programs and services. 

i 

2 

3 

4 

S 

i 

IR staff effectively presents (written and oral) data in ways that are easily understood by readers or audiences. 

i 

2 

3 

4 

S 

j 

Professional development opportunities are made available for IT and IR personnel. 

i 

2 

3 

4 

5 


2.2 

Process for identifying achievement gaps 

a 

College routinely collects, analyzes, and reports longitudinal data on cohorts of students to track student progression and 
outcomes. 

i 

2 

3 

4 

5 

b 

College routinely disaggregates student cohort data by age, race, gender, income, and other factors to identify gaps in 
achievement among student groups. 

i 

2 

3 

4 

5 

c 

College regularly uses qualitative methodologies (e.g., interviews, open-ended surveys, and focus groups) for research with 
students, faculty, and staff to identify strengths and weaknesses in programs and services, as well as opportunities for 
improvement. 

i 

2 

3 

4 

5 


2.3 

Process for formulating and evaluating solutions 

a 

College routinely engages faculty, staff, and others from across the campus community to review data on student achievement 
and help develop and refine strategies for addressing priority problems. 

i 

2 

3 

4 

5 

b 

College routinely evaluates the effectiveness of efforts to improve student success and uses the results to improve policy and 
practice. 

i 

2 

3 

4 

5 

c 

IR staff consults with faculty and other college researchers on program evaluation design, execution, and analysis. 

i 

2 

3 

4 

5 


2.4 

Widespread commitment to and capacity for data-informed decision making 

a 

Organizational structure at the college reinforces strong lines of communication between IR, IT. and executive leadership. 

i 

2 

3 

4 

5 

b 

Executive leaders regularly review data reports and make requests for analyses to IR. 

i 

2 

3 

4 

S 

c 

College leaders and IR personnel copresent research findings at both institutional gatherings and external conferences. 

i 

2 

3 

4 

5 

d 

There are opportunities for college personnel at varying levels to develop questions for research and analyze data 
collaboratively. 

i 

2 

3 

4 

5 

e 

There are multiple college stakeholders involved in reviewing longitudinal and cohort data to identify trends and gaps, 
diagnose problems, and contemplate interventions. 

i 

2 

3 

4 

5 

f 

IT and IR personnel are included in institutional strategic planning conversations. 

i 

2 

3 

4 

S 

g 

Hiring decisions for staff — executive leaders, faculty, IT and IR personnel — consider candidates’ alignment with the 
institution’s commitment to data-informed decision making for student success. 

i 

2 

3 

4 

5 
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Achieving the Dream™ 

Community Colleges Count 

www.AchievingTheDream.org 

Achieving the Dream, Inc. is a national nonprofit organization dedicated to helping more 
community college students, particularly low-income students and students of color, stay in school 
and earn a college certificate or degree. Evidence based, student centered, and built on the 
values of equity and excellence. Achieving the Dream is closing achievement gaps and 
accelerating student success nationwide by 1) guiding evidence-based institutional change; 2) 
influencing public policy; 3) generating knowledge; and 4) engaging the public. Conceived as an 
initiative in 2004 by Lumina Foundation and seven founding partner organizations, today Achieving 
the Dream is the most comprehensive nongovernmental reform movement for student success in 
higher education history. With 160 community colleges and institutions, more than 100 Coaches and 
advisers, and 15 state policy teams working throughout 30 states and the District of Columbia, 
Achieving the Dream helps 3.5 million community college students have a better chance of 
realizing greater economic opportunity and achieving their dreams. 


(f) PUBLIC AGENDA 

Public Agenda, a Founding Partner of Achieving the Dream, was established in 1975 by social 
scientist and author Daniel Yankelovich and former U.S. Secretary of State Cyrus Vance. It works to 
help leaders, stakeholders, and the general public collaborate on sustainable solutions to complex 
issues. Our in-depth research on how citizens think about policy has won praise for its credibility and 
fairness from elected officials of both political parties and from experts and decision makers across 
the political spectrum. Our public engagement team provides technical assistance to leaders in 
communities and states across the nation. Our award-winning website, publicagenda.org, offers 
information about the challenges the country faces and nonpartisan guides to solutions. 
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